Distributing processing for imaging processing

ABSTRACT

Approaches are described for managing the processing of images or video on a computing device. A portable computing device can include one or more dedicated components, such as an application-specific integrated circuit (ASIC) or other dedicated processor component, to be integrated into the computing device to perform at least a portion of the imaging processing of captured images or video. For example, the dedicated processor component can enable the offloading of basic image signal processing, as well as higher level or “machine vision” processing from the device processor of the device. In this way, the dedicated processor component can perform signal processing for which the input is an image (or video), and where image or video data can be analyzed, interpreted and/or manipulated to generate an output, the output of image processing being either an image or a set of characteristics or parameters related to the image. The output can be provided to a device processor for further processing.

BACKGROUND

As computing devices offer increasing processing capacity andfunctionality, users are able to operate these devices in an expandingvariety of ways. For example, computing devices are increasinglyoffering multiple high quality cameras that enable additional types offunctionality. In some devices, these cameras are capable of capturinghigh resolution images and/or videos, and can also be used to providethree-dimensional (3D) image capture for both still and video imaging.As such, these cameras can provide a large amount of imaging data andtherefore present significant opportunities for performing imageanalysis and other computations. However, a downside to providingcameras that can capture high resolution images, and/or are capable of3D image capture, is the fact that processing such images can beexpensive in terms of computation resources and power consumption. Thishas been due in part on the large number of features and components thathave been integrated into the device. Accordingly, manufacturers havebeen looking to improve the battery life and increase the processingperformance of these devices.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments in accordance with the present disclosure will bedescribed with reference to the drawings, in which:

FIG. 1 illustrates an example computing device that includes a dedicatedprocessor component, in accordance with various embodiments;

FIG. 2 illustrates an example computing device that includes a set ofbasic components of a dedicated processor component, in accordance withan embodiment;

FIG. 3 illustrates an example computing device that includes a set ofbasic components of a dedicated processor component, in accordance withan alternate embodiment;

FIG. 4 illustrates an example computing device that includes a set ofbasic components of a dedicated processor component, in accordance withan alternate embodiment;

FIG. 5 illustrates an example process that can be performed by thededicated processor component, in accordance with various embodiments;

FIG. 6 illustrates front and back views of an example portable computingdevice that can be used in accordance with various embodiments;

FIG. 7 illustrates an example set of basic components of a portablecomputing device, such as the device described with respect to FIG. 6;and

FIG. 8 illustrates an example of an environment for implementing aspectsin accordance with various embodiments.

DETAILED DESCRIPTION

Systems and methods in accordance with various embodiments of thepresent disclosure may overcome one or more of the foregoing or otherdeficiencies experienced in conventional approaches for managing theprocessing of images or video via an electronic device. In particular,various embodiments provide approaches for enabling one or morededicated components, such as an application-specific integrated circuit(ASIC) or other dedicated processor component, to be integrated into acomputing device (e.g., a mobile phone, tablet computer, wearablecomputer, etc.) to perform at least a portion of the imaging processingof captured images or video.

For example, the dedicated processor component can enable the offloadingof basic image signal processing, as well as higher level or “machinevision” processing from a general purpose processor or other deviceprocessor of the computing device. In this way, the dedicated processorcomponent can perform image processing for which the input is an image(video or audio), and where image, video, or audio data can be analyzed,interpreted and/or manipulated to generate an output, the output beingeither an image or a set of characteristics or parameters related to theimage, video, or audio. The output can be provided to the generalpurpose processor (or device processor) for further processing. Theoffloading of such tasks can advantageously improve battery life andperformance by utilizing lower power consumption than would otherwise beconsumed if only the device processor were to be utilized. In accordancewith various embodiments, the device processor can be any system on achip (SoC) designed to support applications running in a computingenvironment, such as on a mobile phone, tablet computer or wearablecomputer (e.g., smart classes, smart watch, etc.). In one embodiment,the device processor can be the processor that executes the operatingsystem (OS) and the applications on a computing device.

In certain embodiments, the dedicated processor component can beconnected to or included in a camera. In this situation, the dedicatedprocessor component can be configured to perform specific tasks for aspecific camera. For example, a first dedicated processor componentconnected to or included in a first camera can be configured to performa first task, and a second dedicated processor component connected to orincluded in a second camera can be configured to perform a second task.For example, for the first camera the first dedicated processor canperform head tracking and for the second camera the second dedicatedprocessor component can perform object recognition. In any situation,the dedicated processor component can perform at least a portion of theimage computational processing before providing the processed imageinformation to the device processor, where image computationalprocessing can include image capture, processing, and manipulationtechniques that enhance or otherwise extend the capabilities of image orvideo data, such techniques and approaches further described herein.

In various other embodiments, for example, the dedicated processorcomponent can be shared between any number of cameras. In the situationwhere the dedicated processor component is shared between two or morecameras, the dedicated processor component can be configured toselectively, or alternatively, receive image data from one of thecameras at a time, process (e.g., correct for lens correction, depthpixel correction, format conversation, object identification, etc.)captured image data based on the selected imaging element, and providethe processed image data to a general purpose processor (or deviceprocessor) for further processing (such as to perform advanced objecttracking features, enhance or otherwise modify the image). The multiplecameras can be independently controlled cameras that have differentframe rates and/or different resolutions. Some of these cameras may behigh resolution cameras, some may be medium resolution and yet othercameras may be low resolution cameras. For example, the standardrear-facing digital camera used by a user to obtain a photograph may bea high resolution camera, while other cameras (e.g. front-facing camerasused for gesture detection) may be of lower resolution.

The dedicated processor component can be connected to some or all ofthese cameras by way of a combiner that receives the image data from thecameras over buses capable of carrying image data (e.g., MIPI, SPI,I²C). The dedicated processor component can process all of theinformation coming from the multiple cameras such that the deviceprocessor does not need to be woken up or invoked unless a specifiedevent of interest occurs (e.g., a gesture is detected by the dedicatedprocessor component, a face is recognized, etc.). In some embodiments,two or more of the cameras may be configured as stereo pairs and providethe image data to the dedicated processor component for stereoprocessing. The dedicated processor component can enable rectificationof the images and generate disparity maps for stereo processing and thelike.

As used throughout this disclosure, the terms “machine visioninformation” and “high level information” or “control information” areoften used interchangeably and are intended to cover any information ordata that can be extracted, derived or computed based at least in parton the raw image data received from the one or more cameras to thededicated processor component. By way of some illustrative examples, themachine vision information might include user hand gesture information,facial or object recognition information, head tracking information orother information which was computed based at least in part on one ormore images and which can be used to control a user interface of thedevice. As another example, the machine vision information may includeinformation (e.g., disparity map) obtained from stereo processing two ormore images obtained from cameras that may be configured to function asstereo pairs.

Various other functions and advantages are described and suggested belowas may be provided in accordance with the various embodiments.

FIG. 1 illustrates an example situation 100 wherein a user 102 isinteracting with a computing device 104. Although a portable computingdevice (e.g., a smart phone, an electronic book reader, or tabletcomputer) is shown, it should be understood that various other types ofelectronic device that are capable of determining and processing inputcan be used in accordance with various embodiments discussed herein.These devices can include, for example, notebook computers, personaldata assistants, video gaming consoles or controllers, portable mediaplayers, and wearable computers (e.g., smart watches, smart glasses,etc.) among others. In this example, the computing device 104 includes acamera 106 positioned on a front, side, or corner of the device suchthat the camera will likely be able to capture image information of atleast a portion of the user while the user is viewing content displayedon the device. For example, the camera 106 in FIG. 1 is on the front ofthe device such that an angular capture range 108 of the camera canimage at least a portion of the user while the viewer is viewing contentdisplayed on the display element of the electronic device. In accordancewith the illustrated embodiment, the device further includes arear-facing camera 107 and four corner cameras (120, 122, 124, 126). Byway of illustration, the rear-facing camera 107 can be a high resolutiondigital camera used to take high definition video by the user of themobile device. The front-facing camera 106 can be a medium resolutioncamera designed to optimize video communication (e.g., video conferencecall, face-to-face telephone calls, etc.) between multiple devices overa network. As used throughout this disclosure, a network can be anywired or wireless network of devices that are capable of communicatingwith each other, including but not limited to the cellular networks, theInternet or other Wide Area Networks (WANs), Local Area Networks (LANs),Storage Area Networks (SANs), Intranets, Extranets, and the like.

The device can utilize some of the same elements as a conventionaldevice, such as may include a display screen. The device can alsoinclude high and/or low-resolution cameras that can includeauto-focusing elements for use in still image capture or two-dimensionalvideo capture. The device can include other elements useful for imagingas well, such as a light sensor for determining an amount of ambientlight and a white light LED, or other such illumination element, usefulin illuminating objects within at least a portion of a field of view ofthe camera. Each imaging element may be, for example, a camera includinga complimentary metal-oxide semiconductor (CMOS) device, a motiondetection sensor, a charge coupled device (CCD), an infrared sensor, aquantum dot imager, a gallium arsenide sensor, or any other appropriateimage capturing technology.

In accordance with an embodiment, the four corner cameras (120, 122,124, 126) can be low-resolution cameras designed specifically for userinterface controls (e.g. detecting gestures without touching the mobiledevice, etc.). These cameras can be operating autonomously such that thedevice processor need only be woken up in the event of a particularevent being detected. For example, the device processor may be awakenedupon detecting that the user made a predetermined gesture that wouldcause the device processor to “unlock” the computing device, answer anincoming telephone call, or the like. Further, the cameras can acquirebe used to acquire images and the image information for each image canbe used by the device to determine a relative position and/ororientation of the user with respect to the device. Such imageinformation can also be used for object recognition, object tracking, orvarious other purposes.

However, as mentioned above, in conventional hardware implementations,image computational processing would need to be performed by providingall of the image data from a camera to the device processor overhigh-speed and high power consuming image buses and processing the imagedata at the device processor. For example, image computationalprocessing, such as machine vision algorithms are computationallyintensive and are typically run on the device processor. Thesealgorithms usually contain multiple steps, such as identifying an objectof interest (e.g., a hand, face, head, or some other object) andanalyzing the object to identify features useful in recognizing theobject. In conventional hardware implementations, the device processorcomputes the machine vision algorithms. However, individually or thecombination of the components contributes to the overall powerconsumption. Performing the machine vision algorithms on deviceprocessor, as opposed to a dedicated processor or set of componentsoptimized for a particular algorithm, is expensive in terms of powerconsumption and computing resources.

Accordingly, in accordance with various embodiments, rather thanutilizing these power-costly processes, a dedicated processor componentcan instead perform the majority of the image computational processinglocally and only provide the extracted higher level information (e.g.,machine vision information) to the device processor for furtherprocessing. For example, such image computational processing can includeanalyzing an image acquired by at least one camera to determine an areaof interest. The area of interest can be, for example, a specificportion or object of the image, such as a face, head, hand, foot, etc.The dedicated processor component can define a portion (e.g., a window)of the image that includes the area of interest. The window can range insize, such as 50×50 pixels 100×100 pixels, or any size that includes atleast a portion of the area of interest. The dedicated processorcomponent can provide the area of interest to the device processor forfurther processing (e.g., feature extraction). In this way, the time andpower required to transmit the entire image to the device processor andprocess the entire image can be saved.

In yet another example, image computational processing can includeinforming the device processor whenever it detects that a gesture hasbeen made, or whenever a facial recognition has been detected. In someembodiments, the dedicated processor component can interrupt or “wakeup” the device processor that is in a standby mode (e.g., “sleep” mode)upon detecting that a specific type of machine vision information hasbeen detected. This can significantly reduce the battery powerconsumption, as the device processor can remain in standby mode and doesnot need to continuously obtain all of the image data captured by thecameras and process it. At the same time, if the device processordetermines that further processing of the image data needs to beperformed based on the received machine vision information, it caninitiate the transfer of the image data from the dedicated processorcomponent to the device processor. For example, the device processor cancontinuously inspect the high level information being received from thededicated processor component, determine that the image data may be ofinterest for additional processing based on the received information,and instruct the dedicated processor component to transfer image data tothe device processor as needed. In this manner, the device processor candetermine when image data should be transferred to implementcomputationally intensive algorithms or processes, which can result in asubstantial power savings.

In various embodiments, the dedicated processor component can include animage signal processor (ISP) that performs image computationalprocessing such as low-level image signal correction, and a digitalsignal processor (DSP) that performs image computational processing suchas high-level machine vision processing. Image computational processingsuch as image signal correction performed by the ISP can include actionssuch as dead pixel correction, lens correction, geometric correction,data compression, generating a histogram of the pixel information,automatic exposure control, automatic gain control, automatic whitebalance, de-mosaicing, smoothing, edge enhancement, blob identification,image thresholding, and various other processing. For example, deadpixel correction can refer to any process executed by a processor tocorrect defective, stuck, dead or otherwise malfunctioning pixels on aliquid crystal display (LCD). Lens correction can include any process tocorrect flaws caused by camera lens distortion, such as barrel orpincushion distortion, chromatic aberration and the like. Geometriccorrection can refer to any process for manipulating image data to makethe image's projection match a specific projection surface or shape. Ahistogram of an image can be any representation of pixel intensityvalues of the image. Demosaicing can refer to any process that can beused to reconstruct an image from incomplete samples. Blobidentification can refer to any process that can be used to identifydistinct shapes in an image without actually determining what thoseshapes are. Image thresholding can be any process used to markindividual pixels in an image as object pixels if their value is greaterthan a threshold value (e.g., assuming an object to be brighter than thebackground) and as background pixels otherwise.

Image computational processing such as higher level machine visionprocessing performed by the DSP can include gesture tracking, head/facetracking, object tracking, face/object recognition, multiple imagestereo processing (e.g. image rectification, generating stereo disparitymaps), three dimensional image capture, metadata tagging of the imagedata (e.g., geo-tagging, time-stamping), symbol reading (e.g., barcodes,2D ID codes), optical character recognition (OCR), one or more imagefusion algorithms, sensor fusion algorithms, and the like. Other taskscan include basic level image processing, including additions andsubtractions of image data that can be used for object detection,windowing of a detected object, image segmentation, exposure durationdetermination, ambient light determination, etc. As used herein, machinevision processing can include any process that can be used to analyze animage and extract, deduce or learn information about that image. Forexample, the dedicated processor component can process image data todetermine when a user has made a particular hand gesture (e.g., gesturetracking), detect when a specific face is recognized (e.g., facerecognition), identify that a barcode has been read and provide thevalue of the barcode (e.g., symbol reading), or determine otherinformation that can be provided to the device processor and be used tocontrol the user interface of the device.

In accordance with an embodiment, each dedicated processor component caninclude local memory for storing some or all of the image data receivedfrom a respective camera. This can enable the dedicated processorcomponent to locally process the one or more images withoutreading/writing the data from the main memory of the device. In someembodiments, the dedicated processor component can also have aninterface to the device memory of the device, to which it can write/readthe data if that is deemed necessary. In accordance with variousembodiments, the local memory can be accessed by the DSP to read and/orwrite the images or the machine vision information. In addition, thecomputing device may comprise its own memory which is accessible by thedevice processor and/or the dedicated processors. As described herein,the term memory can encompass any physical device capable of storingdata, such as flash memory, solid state drive (SSD), random accessmemory (RAM), disk based memory and the like.

In accordance with various embodiments, an application programminginterface (API) can be used to control aspects of the dedicatedprocessor component and/or other components of the device. For example,the APIs can be used to control a camera in communication with thededicated processor component, such as to control a camera speed, windowsize of a detected object, image resolution, among others. The APIs canalso be used to specify an object to detect (e.g., hand, face, foot,landscape, animal, etc.) and a portion of the image including thedetected object can be provided to the device processor for furtherprocessing. The dedicated processor component can interface with variousother components as well, such as ambient light sensor, motion detectionsensor (e.g., a gyroscope, an accelerometer, etc.), or any sensor thatcan increase the processing of an image captured by the camera.

FIG. 2 illustrates an example implementation 200 of a computing deviceincluding a dedicated processor component, in accordance with variousembodiments. As shown in FIG. 2, the computing device 201 includescameras 202 and 204, dedicated processor components 206 and 208, asystem or device processor 210, and device memory 212. Although twocameras and two dedicated processor components are shown, these are notto be taken as limiting and more or fewer cameras and/or dedicatedprocessor components are possible. Further, as described, the camerascan include any input device, such as low and high resolution cameras,infrared (IR) cameras, monochrome cameras, motion sensors, ultrasonicsensors, among other input capturing devices. Further still, it shouldbe understood that various other arrangements and combinations can beused as well within the scope of the various embodiments, such as thoseshown in FIG. 3 and FIG. 4.

As shown in FIG. 2, each camera can have a direct or dedicatedconnection to a dedicated processor component. For example, camera 202can be connected to dedicated processor component 206, and camera 204can be connected to dedicated processor component 208. Dedicatedprocessor components 206 and 208 can be connected to the deviceprocessor 210. The device processor can be utilized for various timingcontrol, image processing functions, image computational processingfunctions, and in at least some embodiments can perform additionalprocessing on image or video data captured by the imaging elements,where the additional processing and/or at least some of these actionscan be beyond the capability of the dedicated processor component. Forexample, the in various embodiments, the general purpose processor (ordevice processor) can be configured to perform at least one of gesturedetection, object tracking, object recognition, stereo processing of twoor more images, three dimensional (3D) image capture, image metadatatagging, time-stamping, geographical tagging, symbol reading, depthmapping, image rectification, edge detection, image scaling, objectcharacter recognition, audio recognition, or video recognition.

Dedicated processor components 206 and 208 can include respective imagesignal processors 214 and 216, MP/DSPs 230 and 232 to perform variousimage computational processing approaches, and local memory components234 and 236. In accordance with various embodiments, the dedicatedprocessor components can implement image computational processingfunctions or other functions, and the local memory or other storageelement can be used to store the functions and/or other data such as atleast a frame's (e.g., image) worth of data. For example, in accordancewith an embodiment, each dedicated processor component can analyze theimage data coming in from a respective camera, video data, and/or audiodata. In the situation of image data, the data can be analyzed and thededicated processor component can send small packets of information tothe device processor containing high level information (e.g., machinelevel information). In various embodiments, at least some system-levelcontrol and image computational processing functions can be implementedby the dedicated processor components. Such functions enable a sensorand related components to function as a camera without accessingexternal control circuitry, principally sourcing of clocks to seriallyread out the data including options for decimation (skipping pixels, orgroups of pixels during readout), binning (summing adjacent groups ofpixels), windowing (limiting serial readout to a rectangular region ofinterest), combinations of decimation and windowing, aperture correction(correction of the lens vignetting), and lens correction (correction ofthe lens geometric distortion, at least the radially symmetric portion).Other examples of image computational processing functions include“blob” or region detection for segmenting fingers for hand gestures andface detection and tracking for head gestures. In certain situations,the dedicated processor component is operable to run one or more imagecomputational processes in parallel, wherein the one or more imagecomputational processes include one of gesture detection, objectdetection, object recognition, or image enhancement, or any of the otherimage computational processes described herein. Performing thesefunctions and others on the dedicated processor component enables thedevice processor to remain in standby mode or perform other functions,thereby alleviating the amount of power used by the device processor.

As illustrated, the dedicated processor component includes both an ISPfor performing correction and other low level image processing, and aMP/DSP for performing higher level computations and data analysis on theimage data. The ISP can perform bad pixel correction, relativeillumination correction and de-mosaicing on the images received from thecameras. The microprocessor (MP) or digital signal processor (DSP) canbe configured to perform machine vision processing of the image data. Ina first example, the MP or DSP can analyze an image acquired by at leastone camera to determine an area of interest. The area of interest canbe, for example, a specific portion or object of the image, such as ahead of a user of the computing device, a facial feature of the user, ahand of the user, a foot of the user, an object, audio information, orother information such as character information or video informationamong others. The MP or DSP can define a portion (e.g., a window) of theimage that includes the area of interest. The window can range in size,such as 50×50 pixels 100×100 pixels, or any size that includes at leasta portion of the area of interest. The MP or DSP can provide the area ofinterest to the device processor for further processing (e.g., featureextraction). This can include providing the window of the image, that isthe pixels values including in the window, to the device processor. Invarious other embodiments, the entire image may be provided to thedevice processor as well as coordinates or other information indicatingthe area of interest.

In a second example, the dedicated processor component can be used toprocess gesture and face tracking information. In the illustratedembodiment, the device can be placed on a surface with the displayscreen facing up. In this example, the device can include a number offront-facing corner cameras, as previously described in FIG. 1. When auser makes a gesture in the view of the four cameras, one or more of thecameras can capture the information and provide it to a respectivededicated processor component for processing. For example, a firstcamera may continue capturing images and the dedicated processorcomponent of that camera can process those images to extract the gestureinformation or the like. When the dedicated process detects that apredetermined gesture has been made, it can awaken the device processorby sending the high level information to the device processor. Thedevice processor can in turn inspect the received information and thendecide whether to retrieve the image data from the dedicated processorcomponent for further processing. Similarly, when a user's face movesinto the view of one or more of the cameras (e.g., front-facing camera),the dedicated processor component may process the image to performfacial recognition. For example, if the dedicated processor componentdetermines that the face of the user corresponds to the owner, it mayinvoke the device processor to perform some processing (e.g., unlock thephone, etc.).

The cameras (202 and 204) can be any type of camera that can beintegrated into a computing device. For example, the cameras connectedto a respective dedicated processor component may be high resolution,medium resolution, or low resolution cameras. Some or all of the camerasmay have special modes (e.g., global shuttering, rolling shuttering).Some of the cameras may be monochrome while others may be full color.Some of the cameras may be configured as stereo pairs, while others mayfunction independently of each other. Some cameras may have differentframe rates from other cameras. In accordance with an embodiment, eachcamera feeds into a respective dedicated processor component. It shouldbe noted that although the illustration shows two cameras connected tothe dedicated processor components, the number of cameras should not beperceived as a limitation to all the various embodiments contemplatedherein and that in alternative embodiments, the number of cameras and/ordedicated processor components can include any number.

In accordance with an embodiment, the cameras can be connected to arespective dedicated processor component by way of any link that isappropriate for sending image data. For example, the higher definitioncameras can be connected using a Mobile Industry Processor Interface(MIPI) bus, while the lower resolution cameras may be connected by wayof serial peripheral interface (SPI) or an Inter-Integrated Circuit(I²C) bus. The interface between the dedicated processor components andthe device processor can also be any link, such as a lower power and lowspeed interface such as an I²C bus.

In accordance with various embodiments, the dedicated processorcomponents can access various device sensors 350, such as a cameracontroller and/or any other element, sensor, or controller. These andvarious other components can be controlled or otherwise managed usingthe dedicated processor component and/or device processor using anapplication programming interface (API) or other interface, device orsystem component or service. In various embodiments, various APIs can beused to control aspects of the dedicated processor component as well asthe various other device sensors, components, and/or controllers of thedevice. These device sensors, controllers and/or other components caninclude an ambient light sensor, motion detection sensor (e.g., agyroscope, an accelerometer, etc.), an illumination controller that canbe used to control and/or adjust an amount of illumination used tocapture image or video data, or any sensor that can be controlled oraccessed by the device processor and/or dedicated processor. Further,the APIs can be used to control a camera in communication with thededicated processor component, such as to control a camera speed, windowsize of a detected object, image resolution, among others. Furtherstill, the APIs can be used to specify an object to detect (e.g., hand,face, foot, landscape, animal, etc.) and a portion of the imageincluding the detected object can be provided to the device processorfor further processing. In various embodiments, the device sensor andother components 350 can be controlled by the dedicated processorcomponent and/or the device processor. It should be noted that althoughdevice sensors 350 are shown separate from the general purpose processor(or device processor) and dedicated processor component, in variousembodiments, certain device sensors or at least a portion of theirfunctionality can be included in the dedicated processor component,device processor, and/or distributed between the dedicated processorcomponent and device processor.

In accordance with various embodiments, the camera controller can beused to enable/disable an image or video capture mode, where the cameracontroller can send a signal, or other such communication to thededicated processor component that one of the imaging elements is inoperation. The camera controller can perform other functions, such ascontrol the operation of a lens system such as to control photographiclens functions such as zoom, focus, etc. It should be understood,however, that other elements or controllers such as the illuminationcontroller and the camera controller can alternatively (or additionally)be part of the dedicated circuitry or part of the device processor, andthat other controllers can be used in conjunction with, or used insteadof the illumination and camera controller.

FIG. 3 illustrates an example alternate implementation 300 of acomputing device that includes a dedicated processor component, inaccordance with various embodiments. The structure depicted in thisillustration is similar to the structure depicted in FIG. 2, with somenotable differences. As shown in FIG. 3, the dedicated processorcomponent 306 includes a combiner 308 that is connected to three cameras(302, 303, and 304). The combiner can be any component that receivesmultiple digital or analog input signals (e.g., image data streams) andproduces one or more output signals. The combiner makes it possible formultiple cameras (302, 303, and 304) to share a single dedicatedprocessor component. In some embodiments, the combiner 308 multiplexer(mux) or other such component can sequentially receive one of theincoming signals and provide the selected input to the recipientcomponent (e.g., the device processor 330). In other embodiments, thecombiner can simultaneously provide both signals to the receivingcomponent, combine the signals and/or the like. In other embodiments,the combiner is capable of both (1) switching between the multiplesignals; and (2) combining multiple signals into one resulting signalthat is provided to the device processor 330.

In accordance with various embodiments, camera 302 may be a rear-facinghigh resolution camera integrated into the device, camera 303 may be afront-facing medium resolution camera designed for optimizingface-to-face video communication over a network, and camera 304 may below resolution front-facing camera designed for sensing gestures andother user interface controls. In various embodiments, all of thesecameras may be independently controlled, have different frame rates,etc.

The dedicated processor component includes a processing component 330and local memory 340. As described, the processing component can includean ISP and MP/DSP, as described in regard to FIG. 2. The dedicatedprocessor component can communicate with the device processor 310 suchas by transmitting image data to the device processor. In someembodiments, the dedicated processor component can provide image datafrom certain cameras (e.g., rear-facing high resolution camera) directlyto the device processor so that it can be manipulated, stored in themain device memory 312 or otherwise processed by the device processor.In other embodiments, the device processor may initiate the transfer ofimage data over from the dedicated processor component over a high-speedbus if it determines that further processing is needed, as previouslydescribed. For example, the device processor may periodically receivehigh level machine vision information from the dedicated processorcomponent and analyze the information according to various instructions(e.g., application, operating system, etc.). If the device processordetermines that it needs the original image data to perform furthercomputations on the data, the device processor can cause to betransferred the bulk of the image data from the dedicated processorcomponent to the device processor. In this manner, the dedicatedprocessor component can analyze the images from a number of cameras andinterpret the meanings of those images, all without turning on thedevice processor which may be expensive in terms of power consumption.

In some embodiments, the dedicated processor component can read/writedata directly to the device memory 312, bypassing the device processor.This allows the dedicated processor component to write the image dataand/or the high level machine vision information directly to memory,bypassing the need to provide the data to the device processor. This canprovide improvements for certain applications and can further offloadsome processing from the device processor. For example, a camera (e.g.,a rear-facing camera) can be enabled and the dedicated processorcomponent can receive image data from the rear-facing camera. This canoccur, for example, when a user starts an image capture program on acomputing device, where starting the program can enable the rear-facingcamera. The user can switch or otherwise optionally enable thefront-facing camera by, e.g., selecting an option on the device thatswitches between the rear-facing camera and the front-facing camera, andthe dedicated processor component can then receive image data from thefront-facing camera. Additionally or alternatively, the user can start avideo chat program or other such program that can enable use of thefront-facing camera. In any situation, captured image data from thefront, rear, or another camera can be received and processed by thededicated processor component.

In accordance with various embodiments, a camera can be used to captureimage or video data, and the dedicated processor component can access aconfiguration file to determine how to process captured image or videodata for the camera. For example, the configuration file can includeimage signal processing algorithms and/or settings operable with one ormore imaging elements, where in some instances the signal processingalgorithms are particular to a camera, and in other instances the signalprocessing algorithms can be used with any camera. Thus, based at leastin part on the selected camera, the circuitry can use the configurationfile to perform one or more image signal processing algorithms (such asto process a digital image data or to compress video data), and/or setone or more settings such as a frame rate or exposure time or some otherfunctionality associated with an imaging element.

In accordance with an embodiment, in the situation where multiplecameras are operating at the same time, the combiner, multiplexer (mux),or other such component can be used to select and/or switch between thecameras such that the cameras have access to the dedicated processorcomponent. For example, the combiner can be used to select camera 302,where image or video data captured by the camera 302 is forwarded orotherwise communicated to the dedicated processor component. In thisway, the data captured by the cameras is processed concurrently usingthe dedicated processor component. In other embodiments, the combinercan route image data captured by one of the cameras to the dedicatedprocessor component, and can route image data captured by the otherimaging element to the device processor or other processor separate fromthe dedicated processor component. In this way, the image data beingcaptured by the cameras can be processed concurrently (i.e., some databy the dedicated processor component and other data by the deviceprocessor). In various other embodiments, the combiner can route datacaptured by any one of (or both of) the imaging elements directly to adevice processor, and thus, can bypass the dedicated processorcomponent.

As described, other arrangements are also possible, where a combiner isnot used to route the captured data. In this case, any of (or all of)the cameras can directly communicate captured data to a dedicatedprocessor component, to a device processor, between the dedicatedprocessor component and device processor, or in the situation where eachcamera has dedicated processor component, each camera can communicatecaptured data directly to their respective dedicated processorcomponent.

It should be noted that although the dedicated processor components ofFIG. 2 and FIG. 3 are illustrated as independent components, inalternative embodiments, these components can be integrated into otherelements and structures. For example, the dedicated processor componentscan be integrated into a camera, another processor, and the like. Forexample, FIG. 4 illustrates an example alternate implementation 400 of acomputing device that includes a dedicated processor componentintegrated into a camera, in accordance with various embodiments. Asshown in FIG. 4, a first camera 402 includes dedicated processorcomponent 406 or other control processor that can implement imagecomputational processing functions or other functions. The dedicatedprocessor component includes a processing component 430 and local memory440 to store the functions and/or other data, similar to those describedin regard to FIG. 3. Camera 402 further includes at least one imagingelement 460 configured to capture an image and/or an image. A secondcamera 404 includes dedicated processor component 408. The dedicatedprocessor component includes a processing component 432 and local memory442. Camera 404 further includes at least one imaging element 462.

As described, at least some system-level control and image computationalprocessing functions can be implemented by the dedicated processorcomponent. Such functions enable a sensor and related components tofunction as a camera without accessing external control circuitry,principally sourcing of clocks to serially read out the data includingoptions for decimation (skipping pixels, or groups of pixels duringreadout), binning (summing adjacent groups of pixels), windowing(limiting serial readout to a rectangular region of interest),combinations of decimation and windowing, aperture correction(correction of the lens vignetting), and lens correction (correction ofthe lens geometric distortion, at least the radially symmetric portion).Other examples of on-chip image-processing functions include “blob” orregion detection for segmenting fingers for hand gestures and facedetection and tracking for head gestures. Various other types offunctionality can be provided on the cameras as well in otherembodiments.

Each camera can access a device processor 410, or other such element,and/or any other element or controller such as an illuminationcontroller that can be used to control and/or adjust an amount ofillumination used to capture image or video data. It should beunderstood, however, that other elements or controllers such as theillumination controller and the camera controller can alternatively (oradditionally) be part of the dedicated processor component or part ofthe device processor, and that other controllers can be used inconjunction with, or used instead of the illumination and cameracontroller. In accordance with an embodiment, the device processor canbe utilized for various timing control and image computationalprocessing functions, and in at least some embodiments can performadditional processing on image or video data captured by the imagingelements, where the additional processing and/or at least some of theseactions can be beyond the capability of the dedicated processorcomponent processing of the imaging elements.

For example, the device processor can perform image enhancement, imagerestoration, image compression or other image processing. In accordancewith an embodiment, image enhancement can include accentuation, orsharpening, of image features such as boundaries, or contrast to make agraphic display more useful for display and analysis; image enhancementcan include gray level and contrast manipulation, noise reduction, edgecrispening and sharpening. filtering, interpolation and magnification,pseudo coloring, and so on; and image restoration can include filteringthe observed image to minimize the effect of degradations, and imagecompression can include minimizing the number of bits required torepresent an image.

FIG. 5 illustrates an example process of managing the processing ofimages or video on a computing device, in accordance with variousembodiments. It should be understood that, for any process describedherein, that there can be additional or fewer steps performed in similaror alternative orders, or in parallel, within the scope of the variousembodiments unless otherwise stated. A computing device processor thatincludes at least one camera and an application-specific integratedcircuit (ASIC) or other dedicated processor component can be enabled502. In various embodiments, the device can include a plurality ofcameras and/or dedicated processor components, as well as various othercomponents such as a combiner, an interface, etc. The camera can acquire504 at least one image and the image can be provided to the dedicatedprocessor component. As described, at least a portion of one or moreimage computational processing can be offloaded from a computing deviceprocessor to the dedicated processor component. For example, thededicated processor component can enable the offloading of basic imagecomputational processing that can include image signal processing, aswell as higher level image computational processing that can includemachine vision processing from the device processor of the device. Inthis way, the dedicated processor component can perform imagecomputational processing for which the input is an image (or video), andwhere image or video data can be analyzed, interpreted and/ormanipulated to generate an output, the output of image processing beingeither an image or a set of characteristics or parameters related to theimage.

For example, the dedicated processor component can be configured toanalyze the image to determine 506 an area of interest and define 508 aportion of the image that includes the area of interest. In anotherexample, the dedicated processor component can inform the deviceprocessor whenever it detects that a gesture has been made, or whenevera facial recognition has been detected. In yet another example, in someembodiments, the dedicated processor component can interrupt or “wakeup” the device processor that is in a standby mode (e.g., “sleep” mode)upon detecting that a specific type of machine vision information hasbeen detected. This can significantly reduce the battery powerconsumption, as the device processor can remain in standby mode and doesnot need to continuously obtain all of the image data captured by thecameras and process it. In yet another example, the dedicated processorcomponent can include gesture tracking, head/face tracking, objecttracking, face/object recognition, multiple image stereo processing(e.g. image rectification, generating stereo disparity maps), threedimensional image capture, metadata tagging of the image data (e.g.,geo-tagging, time-stamping), symbol reading (e.g., barcodes, 2D IDcodes), optical character recognition (OCR) and the like.

In response to determining that the processing of the image by thededicated image processor meets a threshold, the dedicated processorcomponent can initiate 510 transfer of the portion of the image to thedevice processor. For example, in various embodiments, the deviceprocessor can determine that the processing of the image by thededicated processor component meets a predetermined threshold, testcriteria, or other threshold metric, and can cause machine visioninformation or other information corresponding to the image to beprovided to the device processor. In this way, when the processing meetsthe threshold, the dedicated processor component is triggered to causethe image information to be provided to the device processor. In variousembodiments, the threshold can be an amount or percentage of processingfor a certain task (e.g., object detection, identification, recognition,etc.). For example, the threshold can represent a certain decision,transition, or break in a decision tree, a number of processing stepspreformed, a minimum or maximum amount of processing time, etc. In someembodiments, this can include detecting a certain object, such as aface, a number of objects, detecting movement of an object, performing apredetermined number of processing steps, etc. In some situations, thethreshold can be changed or altered, can be specific to particularcameras or the same for each camera. In this way, the device processorcan set new, add, remove, or adjust an existing threshold. In thismanner, the device processor can determine when image data should betransferred to implement computationally intensive algorithms orprocesses, which can result in a substantial power savings. At the sametime, if the device processor determines that further processing of theimage data needs to be performed based on the received machine visioninformation, it can initiate the transfer of the image data from thededicated processor component to the device processor. For example, thedevice processor can continuously inspect the high level informationbeing received from the dedicated processor component, determine thatthe image data may be of interest for additional processing based on thereceived information, and instruct the dedicated processor component totransfer image data to the device processor as needed.

When the processing does not meet or otherwise satisfy the threshold,the dedicated processor can perform 512 further processing of the imagebefore initiating transfer of the portion of the image to the deviceprocessor. Further processing can include object detection, imagesegmentation, etc. When the further processing meets the threshold, thededicated processor component can initiate transfer of the portion ofthe image to the device processor.

Thereafter, the device processor can perform 514 at least one imageprocess on the portion of the image. For example, in certainembodiments, upon receiving the portion of the image from the dedicatedprocessor component, the device processor analyzes the image to identifya plurality of feature points; generates, based at least in part uponthe plurality of feature points, a plurality of feature vectorsrepresentative of an object in the image; and identifies the objectbased at least in part on the plurality of feature vectors. In variousembodiments, other processing by the device processor can be performed.Such processing can include advanced object tracking features, enhanceor otherwise modify the image.

FIG. 6 illustrates front and back views of an example electroniccomputing device 600 that can be used in accordance with variousembodiments. Although a portable computing device (e.g., a smartphone,an electronic book reader, or tablet computer) is shown, it should beunderstood that any device capable of receiving and processing input canbe used in accordance with various embodiments discussed herein. Thedevices can include, for example, desktop computers, notebook computers,electronic book readers, personal data assistants, cellular phones,video gaming consoles or controllers, television set top boxes, andportable media players, among others.

In this example, the computing device 600 has a display screen 602(e.g., an LCD element) operable to display information or image contentto one or more users or viewers of the device. The display screen ofsome embodiments displays information to the viewers facing the displayscreen (e.g., on the same side of the computing device as the displayscreen). The computing device in this example can include one or moreimaging elements, in this example including two image capture elements604 on the front of the device and at least one image capture element610 on the back of the device. It should be understood, however, thatimage capture elements could also, or alternatively, be placed on thesides or corners of the device, and that there can be any appropriatenumber of capture elements of similar or different types. Each imagecapture element 604 and 610 may be, for example, a camera, acharge-coupled component (CCD), a motion detection sensor or an infraredsensor, or other image capturing technology.

As discussed, the device can use the images (e.g., still or video)captured from the imaging elements 604 and 610 to generate athree-dimensional simulation of the surrounding environment (e.g., avirtual reality of the surrounding environment for display on thedisplay element of the device). Further, the device can utilize outputsfrom at least one of the image capture elements 604 and 610 to assist indetermining the location and/or orientation of a user and in recognizingnearby persons, objects, or locations. For example, if the user isholding the device, the captured image information can be analyzed(e.g., using mapping information about a particular area) to determinethe approximate location and/or orientation of the user. The capturedimage information may also be analyzed to recognize nearby persons,objects, or locations (e.g., by matching parameters or elements from themapping information).

The computing device can also include at least one microphone or otheraudio capture elements capable of capturing audio data, such as wordsspoken by a user of the device, music being hummed by a person near thedevice, or audio being generated by a nearby speaker or other suchcomponent, although audio elements are not required in at least somedevices. In this example there are three microphones, one microphone 608on the front side, one microphone 612 on the back, and one microphone606 on or near a top or side of the device. In some devices there may beonly one microphone, while in other devices there might be at least onemicrophone on each side and/or corner of the device, or in otherappropriate locations.

The device 600 in this example also includes one or more orientation- orposition-determining elements 618 operable to provide information suchas a position, direction, motion, or orientation of the device. Theseelements can include, for example, accelerometers, inertial sensors,electronic gyroscopes, and electronic compasses.

The example device also includes at least one communication mechanism614, such as may include at least one wired or wireless componentoperable to communicate with one or more electronic devices. The devicealso includes a power system 616, such as may include a battery operableto be recharged through conventional plug-in approaches, or throughother approaches such as capacitive charging through proximity with apower mat or other such component. Various other elements and/orcombinations are possible as well within the scope of variousembodiments.

FIG. 7 illustrates a set of basic components of an electronic computingdevice 600 such as the device 700 described with respect to FIG. 6. Inthis example, the device includes at least one processing unit 702 forexecuting instructions that can be stored in a memory component orelement 704. As would be apparent to one of ordinary skill in the art,the device can include many types of memory, data storage, orcomputer-readable media, such as a first data storage for programinstructions for execution by the processing unit(s) 702, the same orseparate storage can be used for images or data, a removable memory canbe available for sharing information with other devices, and any numberof communication approaches can be available for sharing with otherdevices.

The device typically will include some type of display element 706, suchas a touch screen, electronic ink (e-ink), organic light emitting diode(OLED) or liquid crystal display (LCD), although devices such asportable media players might convey information via other means, such asthrough audio speakers.

As discussed, the device in many embodiments will include at least oneimaging element 708, such as one or more cameras that are able tocapture images of the surrounding environment and that are able to imagea user, people, or objects in the vicinity of the device. The imagecapture element can include any appropriate technology, such as a CCDimage capture element having a sufficient resolution, focal range, andviewable area to capture an image of the user when the user is operatingthe device. Methods for capturing images using a camera element with acomputing device are well known in the art and will not be discussedherein in detail. It should be understood that image capture can beperformed using a single image, multiple images, periodic imaging,continuous image capturing, image streaming, etc. Further, a device caninclude the ability to start and/or stop image capture, such as whenreceiving a command from a user, application, or other device.

The example computing device 700 also includes at least one orientationdetermining element 710 able to determine and/or detect orientationand/or movement of the device. Such an element can include, for example,an accelerometer or gyroscope operable to detect movement (e.g.,rotational movement, angular displacement, tilt, position, orientation,motion along a non-linear path, etc.) of the device 700. An orientationdetermining element can also include an electronic or digital compass,which can indicate a direction (e.g., north or south) in which thedevice is determined to be pointing (e.g., with respect to a primaryaxis or other such aspect).

As discussed, the device in many embodiments will include at least apositioning element 712 for determining a location of the device (or theuser of the device). A positioning element can include or comprise a GPSor similar location-determining elements operable to determine relativecoordinates for a position of the device. As mentioned above,positioning elements may include wireless access points, base stations,etc., that may either broadcast location information or enabletriangulation of signals to determine the location of the device. Otherpositioning elements may include QR codes, barcodes, RFID tags, NFCtags, etc., that enable the device to detect and receive locationinformation or identifiers that enable the device to obtain the locationinformation (e.g., by mapping the identifiers to a correspondinglocation). Various embodiments can include one or more such elements inany appropriate combination.

As mentioned above, some embodiments use the element(s) to track thelocation of a device. Upon determining an initial position of a device(e.g., using GPS), the device of some embodiments may keep track of thelocation of the device by using the element(s), or in some instances, byusing the orientation determining element(s) as mentioned above, or acombination thereof. As should be understood, the algorithms ormechanisms used for determining a position and/or orientation can dependat least in part upon the selection of elements available to the device.

The example device also includes one or more wireless components 714operable to communicate with one or more electronic devices within acommunication range of the particular wireless channel. The wirelesschannel can be any appropriate channel used to enable devices tocommunicate wirelessly, such as Bluetooth, cellular, NFC, or Wi-Fichannels. It should be understood that the device can have one or moreconventional wired communications connections as known in the art.

The device also includes a power system 716, such as may include abattery operable to be recharged through conventional plug-inapproaches, or through other approaches such as capacitive chargingthrough proximity with a power mat or other such component. Variousother elements and/or combinations are possible as well within the scopeof various embodiments.

In some embodiments the device can include at least one additional inputcomponent 718 able to receive conventional input from a user. Thisconventional input can include, for example, a push button, touch pad,touch screen, wheel, joystick, keyboard, mouse, keypad, or any othersuch component or element whereby a user can input a command to thedevice. These I/O components could even be connected by a wirelessinfrared or Bluetooth or other link as well in some embodiments. Somedevices also can include a microphone or other audio capture elementthat accepts voice or other audio commands. For example, a device mightnot include any buttons at all, but might be controlled only through acombination of visual and audio commands, such that a user can controlthe device without having to be in contact with the device.

In some embodiments, a device can include the ability to activate and/ordeactivate detection and/or command modes, such as when receiving acommand from a user or an application, or retrying to determine an audioinput or video input, etc. In some embodiments, a device can include aninfrared detector or motion sensor, for example, which can be used toactivate one or more detection modes. For example, a device might notattempt to detect or communicate with devices when there is not a userin the room. If an infrared detector (i.e., a detector with one-pixelresolution that detects changes in state) detects a user entering theroom, for example, the device can activate a detection or control modesuch that the device can be ready when needed by the user, but conservepower and resources when a user is not nearby.

A computing device, in accordance with various embodiments, may includea light-detecting element that is able to determine whether the deviceis exposed to ambient light or is in relative or complete darkness. Suchan element can be beneficial in a number of ways. In certainconventional devices, a light-detecting element is used to determinewhen a user is holding a cell phone up to the user's face (causing thelight-detecting element to be substantially shielded from the ambientlight), which can trigger an action such as the display element of thephone to temporarily shut off (since the user cannot see the displayelement while holding the device to the user's ear). The light-detectingelement could be used in conjunction with information from otherelements to adjust the functionality of the device. For example, if thedevice is unable to detect a user's view location and a user is notholding the device but the device is exposed to ambient light, thedevice might determine that it has likely been set down by the user andmight turn off the display element and disable certain functionality. Ifthe device is unable to detect a user's view location, a user is notholding the device and the device is further not exposed to ambientlight, the device might determine that the device has been placed in abag or other compartment that is likely inaccessible to the user andthus might turn off or disable additional features that might otherwisehave been available. In some embodiments, a user must either be lookingat the device, holding the device or have the device out in the light inorder to activate certain functionality of the device. In otherembodiments, the device may include a display element that can operatein different modes, such as reflective (for bright situations) andemissive (for dark situations). Based on the detected light, the devicemay change modes.

Using the microphone, the device can disable other features for reasonssubstantially unrelated to power savings. For example, the device canuse voice recognition to determine people near the device, such aschildren, and can disable or enable features, such as Internet access orparental controls, based thereon. Further, the device can analyzerecorded noise to attempt to determine an environment, such as whetherthe device is in a car or on a plane, and that determination can help todecide which features to enable/disable or which actions are taken basedupon other inputs. If voice recognition is used, words can be used asinput, either directly spoken to the device or indirectly as picked upthrough conversation. For example, if the device determines that it isin a car, facing the user and detects a word such as “hungry” or “eat,”then the device might turn on the display element and displayinformation for nearby restaurants, etc. A user can have the option ofturning off voice recording and conversation monitoring for privacy andother such purposes.

In some of the above examples, the actions taken by the device relate todeactivating certain functionality for purposes of reducing powerconsumption. It should be understood, however, that actions cancorrespond to other functions that can adjust similar and otherpotential issues with use of the device. For example, certain functions,such as requesting Web page content, searching for content on a harddrive and opening various applications, can take a certain amount oftime to complete. For devices with limited resources, or that have heavyusage, a number of such operations occurring at the same time can causethe device to slow down or even lock up, which can lead toinefficiencies, degrade the user experience and potentially use morepower.

In order to address at least some of these and other such issues,approaches in accordance with various embodiments can also utilizeinformation such as user gaze direction to activate resources that arelikely to be used in order to spread out the need for processingcapacity, memory space and other such resources.

In some embodiments, the device can have sufficient processingcapability, and the imaging element and associated analyticalalgorithm(s) may be sensitive enough to distinguish between the motionof the device, motion of a user's head, motion of the user's eyes andother such motions, based on the captured images alone. In otherembodiments, such as where it may be desirable for the process toutilize a fairly simple imaging element and analysis approach, it can bedesirable to include at least one orientation determining element thatis able to determine a current orientation of the device. In oneexample, the at least one orientation determining element is at leastone single- or multi-axis accelerometer that is able to detect factorssuch as three-dimensional position of the device and the magnitude anddirection of movement of the device, as well as vibration, shock, etc.Methods for using elements such as accelerometers to determineorientation or movement of a device are also known in the art and willnot be discussed herein in detail. Other elements for detectingorientation and/or movement can be used as well within the scope ofvarious embodiments for use as the orientation determining element. Whenthe input from an accelerometer or similar element is used along withthe input from the camera, the relative movement can be more accuratelyinterpreted, allowing for a more precise input and/or a less compleximage analysis algorithm.

When using an imaging element of the computing device to detect motionof the device and/or user, for example, the computing device can use thebackground in the images to determine movement. For example, if a userholds the device at a fixed orientation (e.g. distance, angle, etc.) tothe user and the user changes orientation to the surroundingenvironment, analyzing an image of the user alone will not result indetecting a change in an orientation of the device. Rather, in someembodiments, the computing device can still detect movement of thedevice by recognizing the changes in the background imagery behind theuser. So, for example, if an object (e.g., a window, picture, tree,bush, building, car, etc.) moves to the left or right in the image, thedevice can determine that the device has changed orientation, eventhough the orientation of the device with respect to the user has notchanged. In other embodiments, the device may detect that the user hasmoved with respect to the device and adjust accordingly. For example, ifthe user tilts their head to the left or right with respect to thedevice, the content rendered on the display element may likewise tilt tokeep the content in orientation with the user.

As discussed, different approaches can be implemented in variousenvironments in accordance with the described embodiments. For example,FIG. 8 illustrates an example of an environment 800 for implementingaspects in accordance with various embodiments. As will be appreciated,although a Web-based environment is used for purposes of explanation,different environments may be used, as appropriate, to implement variousembodiments. The system includes electronic client devices 818, 820,822, and 824, which can include any appropriate device operable to sendand receive requests, messages or information over an appropriatenetwork 804 and convey information back to a user of the device.Examples of such client devices include personal computers, cell phones,handheld messaging devices, laptop computers, set-top boxes, personaldata assistants, electronic book readers and the like. The network caninclude any appropriate network, including an intranet, the Internet, acellular network, a local area network or any other such network orcombination thereof. The network could be a “push” network, a “pull”network, or a combination thereof. In a “push” network, one or more ofthe servers push out data to the client device. In a “pull” network, oneor more of the servers send data to the client device upon request forthe data by the client device. Components used for such a system candepend at least in part upon the type of network and/or environmentselected. Protocols and components for communicating via such a networkare well known and will not be discussed herein in detail. Communicationover the network can be enabled via wired or wireless connections andcombinations thereof. In this example, the network includes theInternet, as the environment includes a Web server 806 for receivingrequests and serving content in response thereto, although for othernetworks, an alternative device serving a similar purpose could be used,as would be apparent to one of ordinary skill in the art.

The illustrative environment includes at least one application server808 and a data store 810. It should be understood that there can beseveral application servers, layers or other elements, processes orcomponents, which may be chained or otherwise configured, which caninteract to perform tasks such as obtaining data from an appropriatedata store. As used herein, the term “data store” refers to anycomponent or combination of components capable of storing, accessing andretrieving data, which may include any combination and number of dataservers, databases, data storage components and data storage media, inany standard, distributed or clustered environment. The applicationserver 808 can include any appropriate hardware and software forintegrating with the data store 810 as needed to execute aspects of oneor more applications for the client device and handling a majority ofthe data access and business logic for an application. The applicationserver provides access control services in cooperation with the datastore and is able to generate content such as text, graphics, audioand/or video to be transferred to the user, which may be served to theuser by the Web server 806 in the form of HTML, XML or anotherappropriate structured language in this example. The handling of allrequests and responses, as well as the delivery of content between theclient devices 818, 820, 822, and 824 and the application server 808,can be handled by the Web server 806. It should be understood that theWeb and application servers are not required and are merely examplecomponents, as structured code discussed herein can be executed on anyappropriate device or host machine as discussed elsewhere herein.

The data store 810 can include several separate data tables, databasesor other data storage mechanisms and media for storing data relating toa particular aspect. For example, the data store illustrated includesmechanisms for storing content (e.g., production data) 812 and userinformation 816, which can be used to serve content for the productionside. The data store is also shown to include a mechanism for storinglog or session data 814. It should be understood that there can be manyother aspects that may need to be stored in the data store, such as pageimage information and access rights information, which can be stored inany of the above listed mechanisms as appropriate or in additionalmechanisms in the data store 810. The data store 810 is operable,through logic associated therewith, to receive instructions from theapplication server 808 and obtain, update or otherwise process data inresponse thereto. In one example, a user might submit a search requestfor a certain type of item. In this case, the data store might accessthe user information to verify the identity of the user and can accessthe catalog detail information to obtain information about items of thattype. The information can then be returned to the user, such as in aresults listing on a Web page that the user is able to view via abrowser on anyone of the user devices 818, 820, 822 and 824. Informationfor a particular item of interest can be viewed in a dedicated page orwindow of the browser.

Each server typically will include an operating system that providesexecutable program instructions for the general administration andoperation of that server and typically will include computer-readablemedium storing instructions that, when executed by a processor of theserver, allow the server to perform its intended functions. Suitableimplementations for the operating system and general functionality ofthe servers are known or commercially available and are readilyimplemented by persons having ordinary skill in the art, particularly inlight of the disclosure herein.

The environment in one embodiment is a distributed computing environmentutilizing several computer systems and components that areinterconnected via communication links, using one or more computernetworks or direct connections. However, it will be appreciated by thoseof ordinary skill in the art that such a system could operate equallywell in a system having fewer or a greater number of components than arcillustrated in FIG. 8. Thus, the depiction of the system 800 in FIG. 8should be taken as being illustrative in nature and not limiting to thescope of the disclosure.

The various embodiments can be further implemented in a wide variety ofoperating environments, which in some cases can include one or more usercomputers or computing devices which can be used to operate any of anumber of applications. User or client devices can include any of anumber of general purpose personal computers, such as desktop or laptopcomputers running a standard operating system, as well as cellular,wireless and handheld devices running mobile software and capable ofsupporting a number of networking and messaging protocols. Such a systemcan also include a number of workstations running any of a variety ofcommercially-available operating systems and other known applicationsfor purposes such as development and database management. These devicescan also include other electronic devices, such as dummy terminals,thin-clients, gaming systems and other devices capable of communicatingvia a network.

Most embodiments utilize at least one network that would be familiar tothose skilled in the art for supporting communications using any of avariety of commercially-available protocols, such as TCP/IP, OSI, FTP,UPnP, NFS, CIFS and AppleTalk. The network can be, for example, a localarea network, a wide-area network, a virtual private network, theInternet, an intranet, an extranet, a public switched telephone network,an infrared network, a wireless network and any combination thereof.

In embodiments utilizing a Web server, the Web server can run any of avariety of server or mid-tier applications, including HTTP servers, FTPservers, CGI servers, data servers, Java servers and businessapplication servers. The server(s) may also be capable of executingprograms or scripts in response requests from user devices, such as byexecuting one or more Web applications that may be implemented as one ormore scripts or programs written in any programming language, such asJava®, C, C# or C++ or any scripting language, such as Perl, Python orTCL, as well as combinations thereof. The server(s) may also includedatabase servers, including without limitation those commerciallyavailable from Oracle®, Microsoft®, Sybase® and IBM®.

The environment can include a variety of data stores and other memoryand storage media as discussed above. These can reside in a variety oflocations, such as on a storage medium local to (and/or resident in) oneor more of the computers or remote from any or all of the computersacross the network. In a particular set of embodiments, the informationmay reside in a storage-area network (SAN) familiar to those skilled inthe art. Similarly, any necessary files for performing the functionsattributed to the computers, servers or other network devices may bestored locally and/or remotely, as appropriate. Where a system includescomputerized components, each such component can include hardwareelements that may be electrically coupled via a bus, the elementsincluding, for example, at least one central processing unit (CPU), atleast one input component (e.g., a mouse, keyboard, controller,touch-sensitive display element or keypad) and at least one outputcomponent (e.g., a display component, printer or speaker). Such a systemmay also include one or more storage components, such as disk drives,optical storage components and solid-state storage components such asrandom access memory (RAM) or read-only memory (ROM), as well asremovable media components, memory cards, flash cards, etc.

Such devices can also include a computer-readable storage media reader,a communications component (e.g., a modem, a network card (wireless orwired), an infrared communication component) and working memory asdescribed above. The computer-readable storage media reader can beconnected with, or configured to receive, a computer-readable storagemedium representing remote, local, fixed and/or removable storagecomponents as well as storage media for temporarily and/or morepermanently containing, storing, transmitting and retrievingcomputer-readable information. The system and various devices alsotypically will include a number of software applications, modules,services or other elements located within at least one working memorycomponent, including an operating system and application programs suchas a client application or Web browser. It should be appreciated thatalternate embodiments may have numerous variations from that describedabove. For example, customized hardware might also be used and/orparticular elements might be implemented in hardware, software(including portable software, such as applets) or both. Further,connection to other computing devices such as network input/outputdevices may be employed.

Storage media and computer readable media for containing code, orportions of code, can include any appropriate media known or used in theart, including storage media and communication media, such as but notlimited to volatile and non-volatile, removable and non-removable mediaimplemented in any method or technology for storage and/or transmissionof information such as computer readable instructions, data structures,program modules or other data, including RAM, ROM, EEPROM, flash memoryor other memory technology, CD-ROM, digital versatile disk (DVD) orother optical storage, magnetic cassettes, magnetic tape, magnetic diskstorage or other magnetic storage components or any other medium whichcan be used to store the desired information and which can be accessedby a system device. Based on the disclosure and teachings providedherein, a person of ordinary skill in the art will appreciate other waysand/or methods to implement the various embodiments.

The specification and drawings are, accordingly, to be regarded in anillustrative rather than a restrictive sense. It will, however, beevident that various modifications and changes may be made thereuntowithout departing from the broader spirit and scope of the invention asset forth in the claims.

What is claimed is:
 1. A computing device, comprising: a computingdevice processor; a camera connected to the computing device processorby a first bus, the camera being configured to capture an image; and anapplication-specific integrated circuit (ASIC) connected to the cameraand the computing device processor by a second bus, wherein at least aportion of image computational processing is performed by the ASIC, theASIC configured to at least: analyze the image to determine an area ofinterest, define a portion of the image that includes the area ofinterest, and in response to determining that image computationalprocessing satisfies a predetermined threshold, cause a transfer of theportion of the image to the computing device processor for furtherprocessing.
 2. The computing device of claim 1, wherein the ASIC furtherincludes: an image signal processor (ISP) configured to perform imagecorrection on the image, the ISP connected to a dedicated image bus thatis capable of providing the image to the computing device processor; anda digital signal processor (DSP) configured to analyze the image todetermine machine vision information, the machine vision informationassociated with at least one of gesture tracking, head tracking, facetracking, object tracking, object recognition, one or more image fusionalgorithms, one or more sensor fusion algorithms, or stereo processingof the image.
 3. The computing device of claim 1, wherein upon receivingthe portion of the image from the ASIC, the computing device processor:analyzes the image to identify a plurality of feature points; generates,based at least in part upon the plurality of feature points, a pluralityof feature vectors representative of an object in the image; andidentifies the object based at least in part on the plurality of featurevectors.
 4. A computing device, comprising: a dedicated processoroperable to be connected to a first camera by a first bus, the dedicatedprocessor configured to identify, from an image acquired using the firstcamera, an area of interest in the image, the dedicated processor beingfurther configured to generate machine vision information based on thearea of interest; and a general purpose processor connected to thededicated processor by a second bus, the general purpose processor beingconfigured to receive, from the dedicated processor and in response toprocessing by the dedicated processor satisfying a predeterminedthreshold, the machine vision information and a portion of the imagecorresponding to the area of interest, the general purpose processorbeing further configured to perform at least one process on the portionof the image corresponding to the area of interest.
 5. The computingdevice of claim 4, further comprising: a combiner configured to receiveimages from a plurality of cameras of the computing device, wherein thededicated processor is capable of processing the images from theplurality of cameras while retaining source information associated witheach image, the source information identifying a camera of the pluralityof cameras that was used to acquire a respective image.
 6. The computingdevice of claim 4, wherein the general purpose processor is furtherconfigured to: analyze the portion of the image corresponding to thearea of interest to identify a plurality of feature points; generate,based at least in part upon the plurality of feature points, a pluralityof feature vectors representative of an object in the image; andidentify the object based at least in part on the plurality of featurevectors.
 7. The computing device of claim 4, wherein the dedicatedprocessor is a first application-specific integrated circuit (ASIC), andwherein the computing device further includes a second camera, the firstcamera being connected to the first ASIC and the second camera beingconnected to a second ASIC, wherein the general purpose processor isconfigured to receive image data from the first ASIC and the secondASIC.
 8. The computing device of claim 4, wherein the dedicatedprocessor is an application-specific integrated circuit (ASIC), andwherein the computing device further includes a second camera, the firstcamera and the second camera being connected to the ASIC, wherein theASIC is configured to receive image data from the first camera and thesecond camera and process a first image from the first camera and asecond image from second camera.
 9. The computing device of 8, whereinthe ASIC is configured to: receive a selection of one of the firstcamera and the second camera; retrieve processing information from aconfiguration file based upon the selection, the processing informationindicating one or more image signal processing algorithms or settings;process an image captured from the selected one of the first camera andthe second camera using the one or more image signal processingalgorithms or settings; and initiate a transfer of the image from theselected one of the first camera and the second camera to the generalpurpose processor.
 10. The computing device of claim 4, wherein thededicated processor further includes: memory storing a configurationfile, the dedicated processor configured to use the configuration fileto perform one or more image signal processing algorithms.
 11. Thecomputing device of claim 4, wherein the dedicated processor includes animage signal processor (ISP) that performs at least one of: dead pixelcorrection, lens correction, geometric correction, data compression,generating a histogram of pixel information, automatic exposure control,automatic gain control, automatic white balance, de-mosaicing, edgeenhancement, image thresholding, image smoothing or image binarization.12. The computing device of claim 4, wherein the general purposeprocessor performs at least one of: gesture detection, object tracking,object recognition, stereo processing of two or more images, threedimensional (3D) image capture, image metadata tagging, time-stamping,geographical tagging, symbol reading, depth mapping, imagerectification, edge detection, image scaling, object characterrecognition, audio recognition, or video recognition.
 13. The computingdevice of claim 4, wherein the dedicated processor is operable to runone or more image computational processes in parallel, and wherein theone or more image computational processes include one of gesturedetection, object detection, object recognition, or image enhancement.14. The computing device of claim 4, wherein the dedicated processor isfurther configured to process at least one of video information andaudio information, and wherein the area of interest includes at leastone of a head of a user of the computing device, a facial feature of theuser, a hand of the user, a foot of the user, an object, audioinformation, character information, or video information.
 15. Thecomputing device of claim 4, wherein at least a portion of machinevision processing is offloaded from the general purpose processor to thededicated processor, thereby at least partly reducing power consumptionof the general purpose processor.
 16. The computing device of claim 4,wherein the first bus and the second bus are electricallyinterconnected.
 17. A method, comprising: receiving an image at adedicated processor from a first camera, the dedicated processor beingconnected to the first camera by a first bus; identifying, by thededicated processor, an area of interest in the image, generating, bythe dedicated processor, machine vision information based on the area ofinterest; determining that processing by the dedicated processorsatisfies a predetermined threshold; and receiving the machine visioninformation and a portion of the image corresponding to the area ofinterest from the dedicated processor at a general purpose processor ofa computing device, the dedicated processor and the general purposeprocessor being connected by a second bus, the general purpose processorbeing configured to perform at least one process on the portion of theimage corresponding to the area of interest.
 18. The method of claim 17,wherein the dedicated processor is an application-specific integratedcircuit (ASIC), wherein the computing device further includes a secondcamera, and the method further comprising: receiving a selection of oneof the first camera or a second camera; retrieving processinginformation from a configuration file based upon the selection, theprocessing information indicating one or more image signal processingalgorithms or settings; processing, by the dedicated processor, an imagecaptured from the selected one of the first camera or the second camerausing the one or more image signal processing algorithms or settings;and imitating a transfer of the processed image to the general purposeprocessor.
 19. The method of claim 17, further comprising: accessing acombiner by the dedicated processor; and selecting, by the combiner,between one of the first camera or a second camera, wherein uponselection of one of the first camera or the second camera, the dedicatedprocessor is configured to process captured data from the selected oneof first camera or the second camera using one or more image signalprocessing algorithms or settings.
 20. The method of claim 17, whereinthe at least one process includes: analyzing the portion of the imagecorresponding to the area of interest to identify a plurality of featurepoints; generating, based at least in part upon the plurality of featurepoints, a plurality of feature vectors representative of an object inthe image; and identifying the object based at least in part on theplurality of feature vectors.
 21. The method of claim 17, wherein thededicated processor is operable to run one or more image computationalprocesses in parallel, and wherein the one or more image computationalprocesses include one of gesture detection, object detection, objectrecognition, or image enhancement.